Skip to content

test(evals): fail invalid-gate-no-number replies that quote the refused figure - #16611

Merged
kyle-sexton merged 2 commits into
mainfrom
fix/16362-invalid-gate-no-number-grader
Nov 16, 2026
Merged

kyle-sexton merged 2 commits into
mainfrom
fix/16362-invalid-gate-no-number-grader

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Closes #16362

Summary

The invalid-gate-no-number case rewarded what the plugin-eval skill forbids.
plugins/evals/skills/plugin-eval/SKILL.md:297 says "Quote no score, delta or interval, even to
refuse it: a quoted figure reads as the result." The case's no-number-from-invalid grader asked
for "Do not post the +0.30", had no FAIL clause for a reply that repeats the figure while refusing
it, and all four pass samples quoted it. A reply that follows the rule and one that breaks it scored
the same.

Fix

  • graders/no-number-from-invalid.md: PASS item 1 asks not to post the delta by reference ("the
    delta", "that number" or "it" meets it). The FAIL list now fails a reply that states the value
    anywhere (+0.30, 0.30, 0.3, .30 or in words), including in a refusal, a quote of the
    user's message, or an explanation of why the figure cannot be trusted.
  • samples/no-number-from-invalid.json: no pass sample contains 0.30. Two new fail samples are
    right on every other point and quote the figure, one in the refusal ("No, don't post the +0.30,
    with or without a footnote") and one in the explanation. The F-partial sample no longer quotes
    it, so it tests only the missing rerun.
  • prompt.md: expected_outcome no longer states the value. The user turn is unchanged.
  • A bump: none changelog fragment for the evals plugin. No skill, script or shipped behavior
    changes.

Done-when items, and what delivers each:

  1. The grader FAILs a reply that states the value, refusal included, and PASS does not ask for it:
    the grader's FAIL clause and PASS item 1 above.
  2. No pass sample contains 0.30: 0 of 4 pass samples do.
  3. A fail sample refuses while quoting the figure: fail samples 5 and 6 (one quotes it in the
    refusal, one in the explanation).
  4. /evals:validate passes: see Verification.
  5. Before and after eval, 6 Sonnet runs, no-number-from-invalid at least 5 of 6 after: the eval
    record below, 6 of 6.

Verification

  • validate-cases.py plugins/evals/evals on the merged head: 0 FAIL. This case's lines are WARNs
    that its sample answers need judge calibration (no model call is made by the check).
  • validate-cases.test.sh: 105 tests OK. check-changelog-fragments.sh --check and
    --check-required origin/main, check-cost-figures.sh --check, markdownlint on the changed
    markdown, and the pre-push check-affected: all pass.

Eval record

Pre-registered before launch: case invalid-gate-no-number alone, --model sonnet, with the
plugin, judge claude-opus-5-5, Claude Code 2.1.325. BEFORE = f6dfd1d5c2 (main's grader), AFTER
= 83aa05c887 (this branch's grader). A 2-run AFTER smoke, then 6 runs per side. The bar is the
issue's (at least 5 of 6 AFTER), used as written. Before launch, the registration estimated only
about a 0.17 chance of meeting it, from 18 stored replies; the owner chose to launch as registered.

The skill text is byte-identical on both sides; only the grader, its samples and expected_outcome
differ. Both sides draw replies from the same distribution, so the compare measures the grader, not
the skill.

  • Smoke: CLEAN. 2 of 2 runs fired, no error, none excluded; every grader passed in both.
  • Compare: MET. Both files valid, 6 launched runs each, all 6 fired and counted on each side,
    no recorded stop.
Grader BEFORE AFTER
no-number-from-invalid 6/6 6/6
why-invalid 6/6 6/6
skill-fired 6/6 6/6
token-keep-temp 6/6 6/6

Figure check (a regex for +0.30, 0.30, 0.3, .30 or "thirty" over each full reply; printed,
not a pass rule):

  • BEFORE: the figure appears in 4 of 6 replies (runs 2, 3, 4 and 6; run 6 twice). Runs 1 and 5
    state none.
  • AFTER: 0 of 6. AFTER runs the judge passed whose reply holds the figure: 0.

What BEFORE shows about the judge grader: main's grader passed all 4 replies that state the figure,
so that grader on its own does not detect the behavior. The figure check is what separates the
sides. The AFTER judge was not shown a reply that quotes the figure in this run, so this record
does not test that it fails one; the two new fail samples are its uncalibrated check (judge
calibration was registered as optional and not run).

Cost: 1.8682 USD across the 14 runs (sum of the result files' costUsd, which already includes
the judge). The run log's running total of 2.3808 adds judgeCostUsd a second time.

Main merged after the eval

main was merged into the branch after the measured head. Against 83aa05c887, the measured
paths (plugins/evals/skills/ and plugins/evals/evals/invalid-gate-no-number/) are unchanged.
Under plugins/evals, main changed only the release version in plugin.json and CHANGELOG.md
(#16537) and lib/prerequisites.{sh,ps1,mjs} (#16519), none of which a run of this case reads.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c

kyle-sexton and others added 2 commits November 16, 2026 01:38
…ed figure

The no-number-from-invalid grader of the invalid-gate-no-number case asked
for "Do not post the +0.30" and had no FAIL clause for a reply that repeats
the figure while refusing it, so it rewarded what plugin-eval SKILL.md:297
forbids ("Quote no score, delta or interval, even to refuse it").

- The grader's PASS item 1 now asks not to post the delta by reference, and
  the FAIL list fails a reply that states the value anywhere, a refusal,
  a quote of the user's message or an explanation included.
- No pass sample holds the figure; two new fail samples are right in every
  other point and quote it, one in the refusal and one in the explanation.
  F-partial no longer quotes it, so it tests only the missing rerun.
- The case's expected_outcome no longer states the value.
- A bump-none changelog fragment for the evals plugin.

Refs: #16362

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 19:20
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T19:22:18.058149Z 139a318 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 139a318288b561bdab7fe991e625ef7eeb2dbafb: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16611

No security issues found. I read every hunk of all 4 changed files.

This PR only changes the evals plugin's invalid-gate-no-number eval case. It also adds a bump: none changelog fragment.

  • graders/no-number-from-invalid.md: the rubric wording changes so a reply now fails if it states the refused delta's value. This is grading text for an eval judge. It is not a standing instruction for any shipped skill, agent, hook or rules file, so the instruction-surface lens finds nothing removed.
  • prompt.md: only expected_outcome changes. allowed_tools: [Read, Glob, Grep, Skill], runs and max_turns are unchanged, so the eval session gets no new tool grant.
  • samples/no-number-from-invalid.json: sample answers are reworded and two fail samples are added. The file is static calibration data.
  • .changes/evals/...md: a changelog fragment with bump: none.

None of the changed files is on the trusted-path list. No plugin manifest, workflow, token request, trusted-actor list, kill switch or merge hold is touched. No code runs differently.

Spend: none beyond prepaid API credit. The PR records 1.87 USD of eval runs already made, and merging changes nothing that bills.

The PR context file has no embedded directives aimed at this lane.

critical-findings: 0

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed 139a318288b561bdab7fe991e625ef7eeb2dbafb: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #16611

This change fixes the eval case invalid-gate-no-number. Its no-number-from-invalid grader now fails a reply that states the delta's value, even when the reply is refusing to post it. That matches the rule in plugins/evals/skills/plugin-eval/SKILL.md ("Quote no score, delta or interval, even to refuse it").

I read all 4 changed files in full, plus the case's sibling grader why-invalid and its samples.

No findings.

Closing issue #16362, done-when items

  1. Grader fails a reply that states the value, refusal included, and PASS does not ask for it: met. PASS item 1 now says "Do not post the delta" and accepts a reference with no value. The FAIL clause names +0.30, 0.30, 0.3, .30 or the value in words, "including while refusing to post it, quoting the user's message, or explaining why it cannot be trusted" (grader L8-L11). The two new fail samples test this clause, but the judge was never calibrated against them. The author says so.
  2. No pass sample contains 0.30: met. A grep of the sample file found the figure on 5 lines, all of them fail samples: F-wrong, F-retract, F-near, F-quote-refuse and F-quote-explain. None of the 4 pass samples contains it.
  3. A fail sample refuses while quoting the figure: met. F-quote-refuse quotes it in the refusal and F-quote-explain quotes it in the explanation. F-partial no longer quotes the figure, so it now tests only the missing rerun advice, and its why was updated to say that.
  4. /evals:validate passes: author-claimed, unverified. The PR reports validate-cases.py with 0 FAIL. I have no shell, so I could not run it.
  5. Before and after eval, at least 5 of 6 after: author-claimed, unverified. The PR records 6/6 after.

Notes (not counted as findings)

  • Outside the closing issue's scope: pass samples in the sibling samples/why-invalid.json still quote the figure ("No +0.30 in the channel", "and no +0.30" at L8 and L12). They are graded only by why-invalid, which does not care about the value, so this change does not break them. Under the updated sibling grader, though, those same replies would now fail. The P-plain sample there already notes that it misses the sibling point, but these two do not.
  • The changelog fragment is bump: none for evals. That fits: only eval case files change, and no shipped skill, script or manifest does. No plugin manifest changed.
  • Spend: the author's eval runs used prepaid API credit (1.87 USD as reported). This change adds no recurring spend.
  • Neither the PR body nor the issue contains embedded instructions aimed at the review.

Verifier dropped 0 of 0 candidates.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 097dd96 Nov 16, 2026
118 of 122 checks passed
@kyle-sexton
kyle-sexton deleted the fix/16362-invalid-gate-no-number-grader branch November 16, 2026 19:59
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(evals): stop the invalid-gate-no-number grader and pass samples from quoting the figure the skill forbids

1 participant